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METHOD AND APPARATUS FOR THREE DIMENSIONALLY 
DISPLAYING EYEGROUND 
AND MEASURING COORDINATES THEREOF 

BACKGROUND OF THE INVENTION 

Technical Field of the Invention 

The present invention relates to methods and apparatus for 
displaying an eyeground three dimensionally and measuring its 
coordinates from photos of the eyeground. 

Description of the Related Art 

An eyeground means the inner portion of an eyeball, and is 
one of the few portions of a human body wherein blood vessels 
and nerves can be directly observed. Conventionally, dedicated 
eyeground cameras have been used to observe the eyeground and 
diagnose abnormalities in blood vessels , retinas , tumors, etc . 

The eyeground camera illuminates the interior of an eye 
through a pupil of a patient, and at the same time the camera 
takes a picture of part of the eyeground on photo film or CCD. 
However, the field of view of the eyeground camera is narrow at 
about 50 degrees, so even a specialized physician cannot easily 
make a precise diagnosis using a photo. 

Therefore, a number of eyeground photos are connected 
according to conventional practice. That is, because the imaging 
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range of an eyeground camera is narrow, photos taken in a number 
of directions such as center, upper, lower, left, and right 
directions are connected together, and the entire eyeground is 
diagnosed. Such connecting work is done not only manually but 
5 also semi automatically using an image processing method. In 
addition, eyeground cameras connected to computers have been 
developed for electronic clinical cards and video images can be 
filed. 

However, because the eyeground is spherical and a photo of 
10 it is a projection of part of the eyeground on a plane, photos 
taken are not continuous and cannot be precisely connected. 

Eyeground photos connected as described above consist of 
planes developed from the spherical eyeground, so only a 
specialist physician can precisely diagnose such photos, 
15 applying extensive expertise of the construction of an eyeball, 
and a physician without specialized knowledge or a patient 
cannot understand the images. 

Furthermore, even if a specialized physician can diagnose 
an abnormality in blood vessels, retinas, tumors, etc. at the 
20 eyeground, images on eyeground photos have already been deformed, 
therefore, sizes and positions of blood vessels, retina 
detachment areas, etc. can neither be quantified as data nor be 
used as time series data or as an epidemic database. 

To solve the aforementioned problems, "Methods and 
25 apparatus for recovering three dimensional patterns of an 

eyeground by superimposing eyeground images taken in multiple 
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directions'" (unexamined Japanese patent publication No. 2002- 
34925) has already been proposed. 

According to this proposal, the lens of a living eye is 
modeled with a single lens, wherein the shape of the eyeground 
5 is regarded to be substantially a sphere; then, an image of the 
sphere taken through the single lens lies on a quadratic surface, 
and it can be assumed that the camera has taken pictures of the 
quadratic surface from various directions. Consequently, as 
shown schematically in Fig. 1, a number of unique points 

10 commonly contained in a number of eyeground photos are 

superimposed on the quadratic surface, thus camera positions and 
poses, when the shape of the quadratic surface and pictures were 
taken and estimated, the parameters of a single lens are 
obtained from the shape of the quadratic surface, and the shape 

15 of the eyeground can be reproduced. 

Nevertherless, actual sizes and shapes of eyeballs vary by 
person, and it is known that if a person who is very near or far 
sighted in particular, the eyeballs deform from a precise sphere. 
Therefore, if the lens of a living eye is modeled with a single 

20 lens and the shape of the eyeground is regarded to be 

substantially a sphere, in the case of a deformed eye ball, if 
any, the reproduced eyeground differs from the actual eyeground, 
so the fact that the reproduced eyeground cannot be used for 
precision diagnostics or a database of eyeground data, is a 

25 problem. 

According to the above mentioned method, if an eye ball is 
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deformed significantly, images cannot be superimposed on a 
virtual quadratic surface. As a result, camera positions and 
poses when each image is photographed can no longer be estimated, 
so parameters for a single lens also cannot be obtained, and 
5 even the shape of the eyeground might not be reproducible. 

SUMMARY OF THE INVENTION 
The present invention aims to solve the aforementioned 
problems. More explicitly, an object of the present invention is 

10 to provide methods and apparatus that even if an eye ball is 
deformed from being a precise spherical surface, a three 
dimensional image of the eyeground can be accurately reproduced, 
and quantitative data on sizes and positions of abnormal blood 
vessels, retina detachment, tumors, etc. can be created from the 

15 above mentioned three dimensional image, and the data can be 
used as time dependent data of the patient and for an epidemic 
database by cubicly displaying the eyeground and measuring 
coordinates three dimensionally . 

The present invention provides methods and apparatus for 

20 three dimensionally displaying an eyeground and measuring the 
coordinates of the eyeground, and is composed of a shape 
measuring step (A) for measuring the size L of an eye ball and 
the shape R of the eyeground, an eyeball setting step (B) for 
setting an eyeball template according to the measured size and 

25 shape, an eyeground image taking step (C) for taking images of 
the eyeground including the superimposed portion d by shifting 
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the imaging position by a predetermined amount h, a parameter 
setting step (D) for obtaining eyeball parameter g, which 
represents the positional relationship between the eyeground and 
a image from the position H of the superimposed portion on the 
5 image, an image pasting step (F) for pasting a number of images 
on the eyeball template according to the eyeball parameter, and 
a three dimensional image display step (H) for displaying the 
three dimensional eyeball image on the eyeball template, onto a 
display device. 

10 According to the above mentioned methods of the present 

invention, because in the shape measuring step (A) , the 
measuring device measures the size L of the eyeball (the length 
from the surface of the cornea to the eyeground) and the shape R 
of the eyeground (radius of curvature at the eyeground) , even if 

15 the eyeball is deformed from being a perfect sphere, an accurate 
eyeball template can be set in the eyeball setting step (B) . 

Because the imaging position is shifted by a predetermined 
amount h and an eyeground including the superimposed portion d 
is photographed in the eyeground image taking step (C) , an 

20 eyeball parameter g that represents the positional relationship 
between the eyeground and the image (for example, CCD of the 
imaging device) can be obtained from the position H of the 
superimposed portion on the image in the parameter setting step 
(D) . 

25 Because imaging positions are shifted each time by a 

predetermined amount h and a number of eyeground areas including 
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superimposed portions d are photographed, and the number of 

images are pasted on the eyeball template based on the eyeball 

parameter g in the image pasting step (F), a three dimensional 

image of the eyeground can be precisely produced, so this image 
5 is displayed in the display device in the three dimensional 

image displaying step (H) to help specialized physicians and 

patients understand the situation. 

According to a preferred embodiment of the present 

invention, there is an image/position recording step (E) in 
10 which an image photographed in the aforementioned eyeground 

image taking step (C) and the positional relationship between 

the eyeball and the imaging device related to the image are 

recorded in a storage medium. 

This method uses the storage medium to store time dependent 
15 data of the patient and provide for an epidemic database. 

The present invention also provides an image positioning 

step (G) in which eyeground images are positioned within the 

eyeground surface according to the respective points at the 

aforementioned superimposed portions d. 
20 Positions of eyeground images can be precisely matched 

using the respective points of papillae, blood vessels, nerves, 

etc. on the eyeground. 

According to another preferred embodiment of the present 

invention, there are a characteristic portion indicating step 
25 (J) for indicating a characteristic portion on the above 

mentioned three dimensional eyeground image displayed in the 
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display device, a three dimensional coordinate defining step (K) 
for obtaining three dimensional coordinates of an indicated 
characteristic portion, a characteristic portion measuring step 
(L) for measuring the size of the characteristic portion, and a 
5 data saving step (M) for saving measured data in the recording 
medium. 

A characteristic portion on a three dimensional eyeground 
image displayed on the display device is indicated in the 
characteristic portion indicating step (J) , thereby the three 

10 dimensional coordinates of the indicated characteristic portion 
can be obtained in the three dimensional coordinate defining 
step (K) ; in addition, the size of the characteristic portion 
can be measured in the characteristic portion measuring step 
(L) ; therefore, based on the three dimensional image, sizes and 

15 positions of pathologic locations such as abnormal blood vessels, 
retina detachment, and tumors can be converted into quantitative 
data . 

In addition, measured data are saved in the storage medium 
in the data saving step (M) , thereby the data can be utilized as 
20 time dependent data of the patient and as an epidemic database. 

The present invention also provides a computer program to 
execute the above mentioned methods , and a storage medium that 
stores the program that the computer can read. 

Using this program and the storage medium, the 
25 aforementioned methods can be implemented with the computer. 

Other objects and advantages of the present invention are 
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revealed in the following description referring to the attached 
drawings . 

BRIEF DESCRIPTION OF DRAWINGS 
5 Fig. 1 is a schematic view of the method of superimposing 

eyeground images in the prior art. 

Figs. 2A and 2B show the positional relationship between an 
eyeball 1 and a imaging device 14 . 

Fig. 3 is a flow chart showing the methods of the present 
10 invention. 

Fig. 4 is another flow chart that shows the methods of the 
present invention . 

Fig . 5 is a coordinate chart of an eyeground ( retina) . 
Fig. 6 is a schematic diagram showing a projection from a 
15 planar image to the eyeground . 

Fig. 7 is a figure that shows the change of a template in a 
projection image. 

Fig. 8A and 8B show example of photo that were shot in 
different directions . 
20 Fig. 9 is the view of an example taking crystalline lens. 

Fig. 10 includes an image in an intermediate tone on a 
display and its sectional view showing an indicated example in a 
specified area. 

Fig. 11 is an image in an intermediate tone on a display 
25 that shows an embodiment of the eyeball template. 

Fig. 12 is an image in an intermediate tone on the display 
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showing an example of an eyeground photo. 

Fig. 13 is an image in an intermediate tone on the display 
showing a three dimensional eyeground image. 

Fig. 14 is an image in an intermediate tone on the display 
5 showing another three dimensional eyeground image. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The present invention is described in more detail referring 
to the drawings. 

10 In each drawing, common portions are identified with the 

same numbers, and no duplicate description is given. 

Figs. 2A and 2B show the positional relationship between 
the eyeball 1 and the imaging device 14 . 

Fig. 2A is a view that also shows the optical system of the 
15 imaging device 14, and Fig. 2B is a schematic view omitting the 

optical system of the imaging device 14. The principles of the 

present invention are described using these figures. 

In Fig. 2A, 14a represents the lens of the imaging device 

14 (eyeground camera), and 14b is a CCD that records images. 
20 When the imaging device 14 takes a picture of the eyeground, the 

following equation (1) is known to hold by omitting the presence 

of the eyeball 1. 

a -1 + b" 1 = f" 1 ... (1) 

Where f is the focal distance of the lens 14a. 
25 Because the eyeball 1 has a refractive index n, which is 

substantially constant, when the size L of the eyeball 1 (length 
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from the surface of retina to the eyeground) is known, the 
equation (2) is valid, where b in equation (1) is replaced with 
b' modified from b by the size L of eyeball 1 and the refractive 
index n. 

a" 1 + b'" 1 = f" 1 ... (2) 

From equation (2) , the ratio of the size of the subject 
(eyeground) and the size of an image 4 on the CCD is shown by 
equation (3), from equation (2). 
Subject: Image = b' : a ... (3) 

In the above, the subject is eyeground 2, which is normally 
a sphere. The image 4 on the CCD lies on a plane. If the 
patient is very near or far sighted, the eyeball 1 deforms from 
being a perfect sphere, but the size L of eyeball 1 (length from 
the surface of the retina to the eyeground) and the shape R of 
eyeground 2 (radius of curvature at the eyeground) can be 
measured using a measuring device 12. 

Therefore, if the image 4 and the relationship between the 
eyeball 1 and the imaging device 14 related to the image (the 
aforementioned a, b, L, and R) are known accurately and the 
image is pasted on the eyeground of the eyeball concerned, a 
three dimensional eyeground image (5) can be displayed on the 
display device of the computer. 

If the crystal lens is taken into account as shown in Fig. 
9, the above mentioned equations (2) and (3) should be modified 
by setting a function that considers the shape of the lens. 

Fig. 3 is a flow chart showing the methods of the present 
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invention, and Fig. 4 is another flow chart showing the same. 

Referring to Fig. 3, the methods of the present invention 
include shape measuring step (A) , eyeball setting step (B) , 
eyeground image taking step (C) , parameter setting step (D) , 
5 image/position recording step (E) , image pasting step (F) , image 
positioning step (G) and three dimensional image displaying step 
(H) . 

In the shape measuring step (A) , the measuring device 12 
measures the size L of eyeball 1 {length from the surface of 
10 retina to the eyeground) and the shape R of eyeground 2 (radius 
of curvature at the eyeground) . 

The size L of eyeball 1 at least includes the length from 
the surface of retina to the eyeground. The shape R of eyeground 
2 also covers at least the radius of curvature at the eyeground. 
15 The measuring device 12 is, for example, an ultrasonic sensor 
that scans at least 3 points per image of an eyeground, or 
irradiates ultrasonic beams to precisely scan the sizes and 
shapes of the eyeball 1 as a whole and the eyeground 2. 

In the eyeball setting step (B) , an eyeball template 3 is 
20 set based on measured size L and shape R. This setting is 

achieved virtually on the computer. That is, the coordinate 
diagram of the eyeground 2 including retina, as shown in Fig. 5, 
is created three dimensionally . 

In the eyeground image taking step (C) , as shown in Figs. 
25 2A, 2B, 8A, and 8B, imaging positions of eyeground 2 are shifted 
each time by a predetermined amount d, and the eyeground 
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including superimposed portions d are photographed a number of 
times . 

In this imaging operation, first a number of images are 
shot at positions slightly displaced from each other, and 
according to deviations between the images, an eyeball parameter 
g that represents the positional relationship between the 
eyeground and a image from position H on the image 4 at a 
superimposed portion is obtained in the parameter setting step 
(D) , For example, as shown in Fig. 2B, if a unique point A of 
the eyeground 2 is displaced by Ah on the image 4 when the 
patient stares at 2 from points T, T' , provided the 2 points T, 
T' are known, the eyeball parameter g, which shows the 
positional relationship between the eyeground and the image, can 
be obtained from the above mentioned equations (1) to (3) . 

In the image/position recording step (E) , image 4 
photographed in the eyeground image taking step (C) and the 
related positional relationship between the eyeball 1 and the 
imaging device 14 are recorded in a recording medium 15, 

In the eyeground image taking step (C) , as shown in Figs. 
8A and 8B in continuation, the imaging positions of the 
eyeground 2 are shifted by a predetermined amount h, and the 
eyeground including superimposed portions D is photographed a 
number of times. Next, as shown in 

Fig. 6, in the image pasting step (F) , a number of images are 
pasted on the eyeball template 3 according to the eyeball 
parameter g. At that time, as shown in Fig. 7, the template on a 
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pro j ected image should be changed to the eyeball template 3 set 
in the eyeball setting step (B) . 

In the image positioning step (G) , the positions of 
eyeground images 8 are adjusted appropriately within the plane 
of the eyeground 2 at the related points 7 in superimposed 
portions d (papilla, blood vessels, nerves, etc. at the 
eyeground) . 

In the three dimensional image displaying step (H) , a three 
dimensional eyeground image 5 on the eyeball template 3 is 
displayed on a display device 16 (for instance, a CRT display) . 
It is also possible to use a three dimensional display, CAVE or 
other devoted VR device (virtual reality device) . The achieved 
three dimensional model can be easily three dimensionally 
visualized on a PC by converting the format to VRML etc. 

According to the aforementioned methods of the present 
invention, in the shape measuring step (A) , the measuring device 
12 measures the size L of eyeball 1 (length from the surface of 
the retina to the eyeground) and the shape R of eyeground 2 
(radius of curvature at the eyeground) , so even if the eyeball 
is deformed from a precision sphere, a precise eyeball template 
3 can be set in the eyeball setting step (B) . 

As the eyeground, including superimposed portions d, is 
photographed by shifting the imaging positions each time by a 
predetermined amount h in the eyeground image taking step (C) , 
an eyeball parameter g that shows the positional relationship 
between the eyeground and a image (for instance, CCD in the 
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imaging device) according to the position H on the image 4 in a 
superimposed portion, can be ascertained in the parameter 
setting step (D) . 

Moreover, in the eyeground image taking step (C) , the 
eyeground including superimposed portions d is photographed by 
shifting the imaging positions each time by the predetermined 
amount h, and a number of images taken are pasted on the eyeball 
template based on the eyeball parameter g in the image pasting 
step (F) , thus a three dimensional image of the eyeground can be 
produced precisely and displayed on a display device 16 in the 
three dimensional image displaying step (H) , to help 
unspecialized physicians and patients understand the clinical 
situation , 

As shown in Fig. 4, the methods of the present invention 
further include three dimensional image displaying step (I) , 
characteristic portion indicating step (J) , three dimensional 
coordinate defining step (K) , characteristic portion measuring 
step (L) and data saving step (M) . 

The three dimensional image displaying step (I) is 
substantially the same as the three dimensional image displaying 
step (H) , wherein a three dimensional image of the eyeground is 
displayed on a display device 16. 

In the characteristic portion indicating step (J) , a 
characteristic portion 9 on the three dimensional eyeground 
image 5 is displayed on the display device 16 (CRT etc.). Such 
an indicateion is given by enclosing the characteristic portion 
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9 on a two dimensional display device such as CRT display, using 
an input device such as a mouse, as shown in Fig. 10. 

In the three dimensional coordinate defining step (K) , 
three dimensional coordinates 10 are obtained from two 
5 dimensional coordinates designated by the characteristic portion 
as indicated. In the characteristic portion measuring step (L) , 
the size of the characteristic portion is measured. These steps 
K and L are processed automatically by the computer. 

In the data saving step (M) , measured data 11 (three 

10 dimensional coordinates 10 and sizes of characteristic portions) 
are saved in a storage medium 15. The storage medium 15 may be 
either tape, drum, hard disk, CD, or any other storage device 
known in the prior art. 

Using any of the above mentioned methods, three dimensional 

15 coordinates 10 of indicated characteristic portions can be 
obtained in the characteristic portion indicating step (J) ; 
furthermore, in that step, the sizes of characteristic portions 
can be measured, and based on three dimensional images, the 
sizes and the positions of pathological portions such as 

20 abnormal blood vessels, retina detachment and tumors can be 
converted into quantitative data. 

By saving measured data (11) to the storage medium (15), 
the data can be used as time dependent data of the patient or in 
an epidemic database. 

25 In addition, the above mentioned methods can be implemented 

by means of a program with the computer and a storage medium 
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that stores the program and can be read by the computer. 

[PREFERRED EMBODIMENTS] 

Preferred embodiments of the present invention are 
5 described below. 

Fig . 11 is an image on a display, showing an embodiment of 
the eyeball template in an intermediate tone. This embodiment is 
an example of a normal eyeball , and the eyeball can be regarded 
forming a spherical surface. In fact, the sizes and the shapes 
10 of eyeballs vary by person; a patient having strong near or far 
sight, is known to have eyeballs that are deformed from a 
perfect sphere. The present invention is characterized in that 
based on the size L of the eyeball 1 and the shape R of the 
eyeground 2, measured in the shape measuring step (A), a precise 
15 eyeball template 3 is set in the eyeball setting step (B) . 

Fig . 12 is an image on a display, showing an example of an 
eyeground photo in an intermediate tone. Obviously, the field of 
view in this photo is narrow at about 50 degrees, so it is 
understood that even a specialized physician cannot easily make 
20 a precise diagnosis from this one photo. 

Figs. 13 and 14 show images on a display, that is, examples 
of three dimensional eyeground images in an intermediate tone. 
Fig. 13 shows a three dimensional eyeground image 5 consisting 
of a number of eyeground photos (9 photos in this example) 
25 pasted on an eyeball template ; actually, the field of view was 
changed in various directions on a computer CRT from inside the 



17 



eyeground, and the eyeground was displayed stereoscopically . 

For Fig. 14, the eyeground image 8 was positioned more 
precisely than in Fig. 13, in the image positioning step (G) ; in 
the same way as for Fig. 13, the field of view can be changed 
freely from inside the eyeball on CRT of a computer, and the 
eyeground can be displayed three dimensionally . 

As described above, according to the present invention, 
information on an eyeground having a three dimensional structure 
is converted into images, that is, two dimensional information, 
when the eyeground of the eyeball is photographed. Although 
these images alone cannot form a three dimensional model of the 
eyeground, the three dimensional model composed uses most of the 
features of the eyeground images. 

In practice, based on eyeground images of a patient and the 
size of an eyeball of the patient, a three dimensional model is 
constructed by virtually projecting images onto the eyeball and 
adding other one dimensional information in the direction of 
depth to the two dimensional images. 

The three dimensional model thus constructed can be 
displayed stereoscopically using a computer graphic (CG) method. 
The model visualized helps to obtain informed consent because 
the patient can understand the construction of the eyeground 
without requiring any special expertise. 

In addition, the three dimensional model thus formed can be 
measured; therefore, based on this three dimensional model, the 
location of damaged blood vessels, range of retina detachment, 
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and tumors can be designated by physician, engineer, or 
automated diagnostic system, and its three dimensional 
coordinates can be ascertained. More explicitly, three 
dimensional coordinates can easily be calculated from the 
5 information on images projected two dimensionally (on the 

monitor) and visualized, by assigning a two dimensional location 
using a mouse etc. 

From the three dimensional coordinates determined, it is 
also possible to define numerical data such as volumes of blood 

10 vessels and retina detached location, and their areas and shapes 
on the retina. With images obtained by a conventional eyeground 
camera, the necessary dimensions cannot be easily measured 
because magnification scales differ between surroundings and 
center of image. However, using the methods of the present 

15 invention, numerical data of such dimensions can be easily 

obtained because a three dimensional model is constructed. These 
numerical data such as for areas can be collated with time 
dependent data of the patient, so therapy criteria such as the 
progress of a disease can be presented. Also, a database can be 

20 constituted by gathering data from many patients, thereby 

epidemic data including diseases (frequently occurring locations, 
relationship with ages) can be provided. Moreover, it is also 
possible to configure basic data for a computer simulated 
analysis of a disease, which is expected to be developed further 

25 in the future. 

The above mentioned methods and apparatus for displaying an 
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eyeground three dimensionally and measuring related coordinates 
enable the precise production of three dimensional images of the 
eyeground, even if the eyeball is deformed from being a perfect 
sphere, so the methods and the apparatus provide great 
advantages such as presenting quantitative data on sizes and 
locations of pathologic locations of abnormal blood vessels, 
retina detachment, tumors, etc. based on the aforementioned 
three dimensional images, and use the data for time dependent 
therapy on the patient and in an epidemic database. 

Although the present invention has been described referring 
to several preferred embodiments, it should be understood that 
the scope of rights covered by the present invention is not 
limited only to these embodiments. Conversely, the scope of 
rights of the present invention should include all modifications, 
corrections, and similar entities included in the claims 
attached. 



